Add support for aarch64 #2973

rambleraptor · 2026-01-27T19:49:48Z

Rationale for this change

This should allow us to build aarch64 wheels on Linux. We have to use QEMU since GitHub doesn't have arm64 Linux runners (just Mac).

Are these changes tested?

Testing CI is awful! We shouldn't merge this in until after 0.11 and then see what happens.

Are there any user-facing changes?

.github/workflows/pypi-build-artifacts.yml

kevinjqliu · 2026-01-27T23:32:49Z

the nightly build ci allows adhoc trigger for a specific branch
we can test this there

geruh · 2026-01-28T04:42:15Z

I noticed that the original issue mentioned QEMU being slow. I looked at how other projects handle this and saw that Polars recently added native ARM runners here: pola-rs/polars#25894

It looks like Github recently (good timing lol...) added ubuntu-24.04-arm for free on public repos, so maybe we can just add this to the matrix and skip QEMU.

https://github.blog/changelog/2025-01-16-linux-arm64-hosted-runners-now-available-for-free-in-public-repositories-public-preview/

rambleraptor · 2026-01-28T04:54:06Z

Interesting! Their public docs don't mention them, so I thought QEMU was the way to go.

This is absolutely perfect! I'll update this tomorrow. Bonus, we'll get to see if the test suite fails at all on ARM (not expecting it to since we have Mac CI)

geruh · 2026-01-28T06:51:45Z

.github/workflows/svn-build-artifacts.yml

-          CIBW_ARCHS: "auto64"
+          CIBW_ARCHS: "auto64 aarch64"
          CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.14"
+          CIBW_MUSLLINUX_X86_64_IMAGE: "musllinux_1_2"


These configs are set to musllinux_1_2 by default, got a successful run without them.

geruh · 2026-01-28T07:05:09Z

Yeah it's super cool! The comparison helped uncover some other things with our release too. But I did some testing on my fork with these changes, and ran a nightly build with this and it worked pretty well!

Nightly run: https://github.com/geruh/iceberg-python/actions/runs/21427296271
PyPi test env deploy: https://test.pypi.org/project/pyiceberg-geruh/

The change becomes much simpler and we can add ubuntu-24.04-arm to the matrix. I'll paste the patch below against main (1f947a7) for you check out!

Subject: [PATCH] Add Linux aarch64 wheel builds using native ARM runners
---
Index: .github/workflows/pypi-build-artifacts.yml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/.github/workflows/pypi-build-artifacts.yml b/.github/workflows/pypi-build-artifacts.yml
--- a/.github/workflows/pypi-build-artifacts.yml	(revision 1f947a72f65632e5f6167ca9d56cef215906362a)
+++ b/.github/workflows/pypi-build-artifacts.yml	(revision 72f3daca16dc072ce9becb8662abb2ec396662a5)
@@ -32,7 +32,7 @@
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [ ubuntu-latest, windows-latest, macos-15-intel, macos-latest ]
+        os: [ ubuntu-latest, ubuntu-24.04-arm, windows-latest, macos-15-intel, macos-latest ]
 
     steps:
       - uses: actions/checkout@v6
@@ -59,7 +59,7 @@
       # the repository, otherwise the tests will fail
       - name: Compile source distribution
         run: uv build --sdist
-        if: startsWith(matrix.os, 'ubuntu')
+        if: matrix.os == 'ubuntu-latest'
 
       - name: Build wheels
         uses: pypa/cibuildwheel@v3.3.1
@@ -77,7 +77,7 @@
 
 
       - name: Add source distribution
-        if: startsWith(matrix.os, 'ubuntu')
+        if: matrix.os == 'ubuntu-latest'
         run: ls -lah dist/* && cp dist/* wheelhouse/
 
       - uses: actions/upload-artifact@v4
Index: .github/workflows/svn-build-artifacts.yml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/.github/workflows/svn-build-artifacts.yml b/.github/workflows/svn-build-artifacts.yml
--- a/.github/workflows/svn-build-artifacts.yml	(revision 1f947a72f65632e5f6167ca9d56cef215906362a)
+++ b/.github/workflows/svn-build-artifacts.yml	(revision 72f3daca16dc072ce9becb8662abb2ec396662a5)
@@ -32,7 +32,7 @@
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [ ubuntu-latest, windows-latest, macos-15-intel, macos-latest ]
+        os: [ ubuntu-latest, ubuntu-24.04-arm, windows-latest, macos-15-intel, macos-latest ]
 
     steps:
       - uses: actions/checkout@v6
@@ -54,7 +54,7 @@
       # the repository, otherwise the tests will fail
       - name: Compile source distribution
         run: uv build --sdist
-        if: startsWith(matrix.os, 'ubuntu')
+        if: matrix.os == 'ubuntu-latest'
 
       - name: Build wheels
         uses: pypa/cibuildwheel@v3.3.1
@@ -71,7 +71,7 @@
           CIBW_SKIP: "cp3*t-*"
 
       - name: Add source distribution
-        if: startsWith(matrix.os, 'ubuntu')
+        if: matrix.os == 'ubuntu-latest'
         run: ls -lah dist/* && cp dist/* wheelhouse/
 
       - uses: actions/upload-artifact@v4

rambleraptor · 2026-01-28T21:27:37Z

@geruh thanks for the investigation! I was really hoping it could be that simple of a fix before I went down the QEMU rabbit hole.

Fokko

Thanks @rambleraptor for fixing this, and thanks @geruh for the thorough review. Do we feel confident to add this for the 0.11 release? I think it would be super valuable since multiple users were asking for this.

rambleraptor · 2026-01-28T21:45:10Z

I'm confident since we aren't changing the CI path for any other platforms.

Worst case (and I don't expect this), this would impact just the release for linux arm64

geruh · 2026-01-28T22:10:53Z

I just ran a test against the wheels in test pypi. It installed fine in Docker on both manylinux and musllinux aarch64 containers, and it seems like everything is working properly.

I didn't test the full suite on ARM, but the CI already runs decoder tests on the native ARM runner and those passed. That being said I believe this is ready for 0.11.0. Now whether or not we use native runners or QEMU, the wheels should have same behavior!

kevinjqliu · 2026-01-29T02:12:06Z

i ran the nightly build for this branch: https://github.com/apache/iceberg-python/actions/runs/21462770268

I see ubuntu-24.04-arm built these wheels, which includes aarch64 🥳

Wheel	Size	Build identifier	Time	SHA256
pyiceberg-0.10.0.dev20260129020217-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl	708.0 kB	cp310-manylinux_aarch64	a minute	12c6bb9227860212c9f5b3ad43657ed304dc517427cf972ee38e5d12e2c30c08
pyiceberg-0.10.0.dev20260129020217-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl	722.2 kB	cp311-manylinux_aarch64	23 seconds	d509fe0316bd60795a5536e6eeb4c6931022aa5abd9b9ad889528abf563edba0
pyiceberg-0.10.0.dev20260129020217-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl	728.9 kB	cp312-manylinux_aarch64	16 seconds	2422b3bc93d9f52f239d61297bb71f343ba05fffb3deb918bc089ca8b649c99b
pyiceberg-0.10.0.dev20260129020217-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl	721.6 kB	cp313-manylinux_aarch64	17 seconds	4839c57125d5f09741e2d7c6d3e17faaa58963be6d6dedd6872daec9ad1034bc
pyiceberg-0.10.0.dev20260129020217-cp310-cp310-musllinux_1_2_aarch64.whl	703.5 kB	cp310-musllinux_aarch64	59 seconds	8245cc4995b6c3c96b3e86431ca4cd582ce4d850cf1b4c1f09255454c8ccfcce
pyiceberg-0.10.0.dev20260129020217-cp311-cp311-musllinux_1_2_aarch64.whl	717.3 kB	cp311-musllinux_aarch64	24 seconds	84550826636ed6940f1216c3fab450c64eced66ee126a5ae4445738117dd0335
pyiceberg-0.10.0.dev20260129020217-cp312-cp312-musllinux_1_2_aarch64.whl	723.4 kB	cp312-musllinux_aarch64	19 seconds	d1674798d82dbf670ae43c9a68791438a5296395a598530829bcfb998b2a186f
pyiceberg-0.10.0.dev20260129020217-cp313-cp313-musllinux_1_2_aarch64.whl	715.4 kB	cp313-musllinux_aarch64	19 seconds	4398e563a5d1d68a32b3aee27251b14374c373ae7294ab7bf8376598ad7db8f3

kevinjqliu · 2026-01-29T02:12:36Z

Thanks for the PR @rambleraptor and thanks for the review @Fokko @raulcd @geruh

raulcd reviewed Jan 27, 2026

View reviewed changes

.github/workflows/pypi-build-artifacts.yml Outdated Show resolved Hide resolved

geruh reviewed Jan 28, 2026

View reviewed changes

Fokko modified the milestone: PyIceberg 0.11.0 Jan 28, 2026

arch changes

13b4694

rambleraptor force-pushed the aarch64 branch from a3a07f5 to 13b4694 Compare January 28, 2026 21:26

Fokko added this to the PyIceberg 0.11.0 milestone Jan 28, 2026

Fokko approved these changes Jan 28, 2026

View reviewed changes

kevinjqliu deployed to testpypi January 29, 2026 02:09 — with GitHub Actions View deployment

kevinjqliu merged commit b11d865 into apache:main Jan 29, 2026
10 checks passed

kevinjqliu mentioned this pull request Jan 29, 2026

Provide pre-built wheels #2970

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for aarch64 #2973

Add support for aarch64 #2973

rambleraptor commented Jan 27, 2026

Uh oh!

Uh oh!

kevinjqliu commented Jan 27, 2026

Uh oh!

geruh commented Jan 28, 2026 •

edited

Loading

Uh oh!

rambleraptor commented Jan 28, 2026

Uh oh!

geruh Jan 28, 2026

Uh oh!

geruh commented Jan 28, 2026

Uh oh!

rambleraptor commented Jan 28, 2026

Uh oh!

Fokko left a comment

Uh oh!

rambleraptor commented Jan 28, 2026

Uh oh!

geruh commented Jan 28, 2026

Uh oh!

kevinjqliu commented Jan 29, 2026

Uh oh!

Uh oh!

kevinjqliu commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Add support for aarch64 #2973

Add support for aarch64 #2973

Conversation

rambleraptor commented Jan 27, 2026

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

kevinjqliu commented Jan 27, 2026

Uh oh!

geruh commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rambleraptor commented Jan 28, 2026

Uh oh!

geruh Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

geruh commented Jan 28, 2026

Uh oh!

rambleraptor commented Jan 28, 2026

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

rambleraptor commented Jan 28, 2026

Uh oh!

geruh commented Jan 28, 2026

Uh oh!

kevinjqliu commented Jan 29, 2026

Uh oh!

Uh oh!

kevinjqliu commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

geruh commented Jan 28, 2026 •

edited

Loading